Motion compensated inter-frame coding has been widely adopted in various coding standards, such as MPEG-1/2/4 and H.261/H.263/H.264/AVC. Motion estimation and compensation as well as subsequent processing in such coding systems are performed on a block basis. During compression process, coding noises may arise due to lossy operations applied such as quantization. The coding artifacts may become noticeable in the reconstructed video data, especially at or near block boundaries. In order to alleviate the visibility of coding artifacts, a technique called deblocking has been used in newer coding systems such as H.264/AVC and the High Efficiency Video Coding (HEVC) system being developed. The deblocking process applies filtering across block boundaries adaptively to smooth the large transitions at and near block boundaries due to coding noises while retaining image sharpness. Furthermore, due to the nature of inter-frame coding, the deblocking process is configured for in-loop operation. In the recent HEVC development, adaptive loop filtering (ALF) is being adopted to process reconstructed frames or deblocked reconstruction frames. Adaptive loop filtering is used as in-loop processing in addition to deblocking and is often applied after deblocking of reconstructed video data. The filter coefficients may be designed according to a known optimization algorithm such as Wiener-Hopf equation that minimizes mean square errors between original frames and reconstruction frames.
In the HEVC system, the conventional ALF can be selectively turned ON or OFF for each block in a frame or a slice. The block size and block shape can be adaptive, and the information of block size and block shape can be explicitly sent to decoders or implicitly derived in a decoder. According to a selected performance criterion, determination will be made for each block regarding whether the block is subject to ALF or not. In U.S. patent application Ser. No. 13/093,068, entitled “Method and Apparatus of Adaptive Loop Filtering”, filed Apr. 25, 2011, an advanced ALF is disclosed to allow a choice of multiple filter sets that can be applied to the reconstructed video data adaptively. Furthermore, the advanced ALF adopts a new unit for the filtering process to apply so that more flexible and/or localized processing can be provided. While the U.S. patent application Ser. No. 13/093,068 discloses flexible adaptive loop filters, it is desirable to develop a system incorporating advanced in-loop filter that extends FU (filter unit) to general processing structure (named to-be-filtered region in this disclosure) for in-loop filter and extend ALF to including other in-loop filters. Furthermore, it is desirable to incorporate rate-distortion optimization procedure to optimize system performance among various processing structures and various candidate filters.